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(57) ABSTRACT 

Disclosed are methods and apparatuses provided for modify- 
ing the effect of an operator controlled input device on an 
interactive device to encourage the self-regulation of at least 
one physiological activity by a person different than the 
operator. The interactive device comprises a display area 
which depicts images and apparatus for receiving at least one 
input from the operator controlled input device to thus permit 
the operator to control and interact with at least some of the 
depicted images. One effect modification comprises mea- 
surement of the physiological activity of a person different 
from the operator, while modifying the ability of the operator 
to control and interact with at least some of the depicted 
images by modifying the input from the operator controlled 
input device in response to changes in the measured physi- 
ological signal. 
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TEAM ELECTRONIC GAMEPLAY 
COMBINING DIFFERENT MEANS OF 
CONTROL 

CROSS-REFERENCE TO RELATED 
APPLICATION 

This patent application is a continuation-in-part of U.S. 
patent application Ser. No. 11/536,811, filed Sep. 29, 2006, 
now issued as U.S. Pat. No. 8,062,129. This application is 
related to commonly owned U.S. application Ser. No. 13/1 66, 
166, entitled PHYSIOLOGICALLY MODULATING 
VIDEOGAMES OR SIMULATIONS WHICH USE 
MOTION-SENSING INPUT DEVICES, and to commonly 
owned U.S. application Ser. No. 61/499,733, entitled 
METHOD AND SYSTEM FOR PHYSIOLOGICALLY 
MODULATING VIDEOGAMES WHICH USE HAND 
AND BODY MOTION-SENSING DEVICES, both of which 
are filed concurrently, the entire contents of both of which are 
incorporated herein by reference in its entirety. This patent 
application claims the benefit of U.S. Provisional Patent 
Application No. 61/361,086, filed Jul. 2, 2010. 

STATEMENT REGARDING FEDERALLY 
SPONSORED RESEARCH AND DEVELOPMENT 

This invention was made in part with Government support. 
The Government may have certain rights in this invention. 

FIELD OF THE INVENTION 

1'he present invention relates generally to the use of bio- 
feedback to modify a subject’s behavior, mental state, and/or 
physiological functioning. More specifically, the invention 
relates to apparatus and methods for modulating an operator’ s 
control input to an electronic game or simulator in response to 
measured physiological activity, such as autonomically-me- 
diated and/or EEG physiological activity, wherein the player 
thereby leams to control the physiological and/or EEG activ- 
ity- 

BACKGROUND 

Biofeedback systems can be used for a variety of purposes, 
such as to address behavioral disorders, such as Attention 
Deficit Disorder, and for training job-related physiological 
activity. 

In brainwave biofeedback training, for example, the trainee 
is typically provided information in the fonn of a convention- 
ally produced electroencephalograph (EEG) display which 
shows him how much he is producing the brainwave 
pattem(s) indicative of attention and/or inattention. This dis- 
play is typically in bland, minimally motivating fomiats, and 
trains individuals by focusing their attention directly on the 
status of targeted physiological signals. For example, this 
feedback frequently consists of a video representation of the 
EEG graph. The procedure, though providing useful informa- 
tion, is often very limited in variation and predictable, and 
accordingly can induce boredom. This can lead to frustration 
when progress is slow, and makes it hard to encourage simul- 
taneous desirable changes in multiple physiological param- 
eters due to limitations in ability to attend to multiple signals. 
Positive reinforcement of attention states can accordingly be 
difficult to obtain, especially in children. 

Additionally, with increased sophistication in technology, 
human performance has increasingly become an important, 
frequently limiting, factor in the proper performance of many 
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advanced technology job-related tasks. For example, both 
inattention and stress overload can play a substantial role in 
impairing pilot performance and producing flight hazards. 
Current biofeedback methods are hard to apply to job-related 
5 physiological training because the necessary focus on physi- 
ological feedback signals can distract trainees from challeng- 
ing professional tasks. The ability to control physiological 
activity, such as to control stress or to remain aware of fluc- 
tuating attentional states, the ability to maintain effective 
to physiological states, and the ability to recover efficiently from 
attention lapses or other ineffective physiological states are 
valuable in task settings requiring recognition and response. 

U.S. Pat. No. 5,377,100 to Pope et al. entitled “Method of 
Encouraging Attention by Correlating Video Game Difficulty 
15 with Attention Level” demonstrates the concept of improving 
attention skill by rewarding specific brain signal patterns with 
success at playing an action video game. The game is virtually 
impossible to win until the player exhibits the required brain 
signal patterns that accompany normal vs. attention-deficit 
20 behavior. Once the player exhibits the required “normal” 
brain signal patterns the game becomes manageable. A mea- 
surement system senses EEG signals and routes them to the 
computer where the game difficulty control sinal is derived. 
This invention has the disadvantage of requiring extensive 
25 reprogramming of a video game, orthe complete construction 
of a new video game, in order to implement the method. 
Because much video game software is proprietary and/or not 
available in source code, this software would be unusable for 
implementing the method. 

30 Some commercial products, for example, “The Mind 
Drive,” sense physiological signals and use the signals alone 
to drive a video game. Because products like “The Mind 
Drive” do not deliver biofeedback training while the trainee is 
playing a game or performing a task in a conventional way, 
35 this method does not reinforce desirable physiological 
changes in the realistic context of task performance. Products 
like Mind Drive, too, have the disadvantage of requiring 
extensive video game programming in order to implement the 
method. 

40 U.S. Pat. No. 6,450,820 to Pope et al. (incorporated by 
reference herein in its entirety) modulates an electronic game 
controller using the physiological signals being produced by 
the player who is manually operating the game controller. 
This method does not translate well to the situation where a 
45 player is very physically active while operating the game 
controller, such as the Nintendo Wii remote controller (Nin- 
tendo and Wii are registered trademarks of Nintendo of 
America, Inc.), because much movement may disrupt or con- 
found the sensing of physiological signals. The prior art 
50 method also does not encompass the situation where a com- 
bination of physical game controller input and physiological 
input from two (or more) different players acting coopera- 
tively combines to affect game performance. 

55 BRIEF SUMMARY OF THE INVENTION 

In one embodiment of the invention, a method is provided 
for modifying the effect of an operator controlled input device 
on an interactive device to encourage the self-regulation of at 
60 least one physiological activity by a person different than the 
operator. The interactive device comprises a display area 
which depicts images and apparatus for receiving at least one 
input from the operator controlled input device to thus permit 
the operator to control and interact with at least some of the 
65 depicted images. The method for modifying comprises the 
steps of measuring at least one physiological activity of a 
person different than the operator to obtain at least one physi- 
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ological signal having a value indicative of the level of the at 
least one physiological activity, and modifying the ability of 
the operator to control and interact with at least some of the 
depicted images by modifying the input from the operator 
controlled input device in response to changes in the at least 
one physiological signal. 

In addition to the method for modifying the effect of an 
operator controlled input device on an interactive device to 
encourage the self-regulation of at least one physiological 
activity by a person different than the operator, as described 
above, other aspects of the present invention are directed to 
corresponding apparatus for modifying the effect of an opera- 
tor controlled input device on an interactive device to encour- 
age the self-regulation of at least one physiological activity by 
a person different than the operator. 

Additional objects, embodiments and details of this inven- 
tion can be obtained from the following detailed description 
of the invention. 

DETAILED DESCRIPTION OF THE INVENTION 

Previous work by the present inventors has been published 
as U.S. Patent Application No. 2008/0081692 Al, which is 
herein incorporated by reference thereto in its entirety. 

The present invention now will be described more hilly 
hereinafter, in which preferred embodiments of the invention 
are discussed. This invention may, however, be embodied in 
many different forms and should not be construed as limited 
to the embodiments set forth herein; rather, these embodi- 
ments are provided so that this disclosure will be thorough 
and complete, and will fully convey the scope of the invention 
to those skilled in the art. 

The present invention includes a method of transforming 
physiological information obtained from biomedical instal- 
ments in order to use that information to modify the function- 
ing of computer simulation or game controllers or joysticks. 
The invention involves modulation that transforms the con- 
troller signals received at a computer’s game port or a game 
console prior to their being used by the computer simulation 
or game software. The result can be that the magnitude of the 
effect of the game or simulation’ s input device (e.g., joystick, 
game pad, steering wheel) is modulated by the strength of the 
physiological signal(s). By making the joystick’s “control 
authority” proportional to the physiological signal(s), the 
player is encouraged to change the physiological signals) 
according to a programmed criterion (e.g., increase, decrease, 
or maintain) in order to perform better at the game task. When 
the physiological signal(s) are the target of physiological 
self-regulation or biofeedback training, the game play rein- 
forces therapeutic changes in related physiological processes. 
However, the reinforcing feedback is preferably implicit in 
the task, and not explicit in the form of direct feedback (bar 
graphs, tracings), as in conventional biofeedback training. In 
this way, contingencies for subtle conditioning of the desir- 
able physiological response(s) are set up. 

Different embodiments of the present invention are pos- 
sible, and the components of the invention can vary depend- 
ing upon implementation. Further, the invention is intended to 
be used with a variety of systems, such as standard video 
game systems (e.g., Sony Playstation, Nintendo, etc.), with 
standard personal computer systems, and with computer 
simulators and professional job training systems. Addition- 
ally, one or more of a wide variety of different measured 
physiological signals can be used in accordance with the 
present invention, for example, EEGs, skin temperature, skin 
conductance, heart rate, and/or event-related potentials 
(ERPs). U.S. Pat. No. 5,377,100, issued on Dec. 27, 1994 to 
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Pope et al, and which is incorporated herein by reference as if 
set forth in its entirety, and also specifically at column 3, line 
8 to column 5, line 60 that details a method for determining an 
individual’s EEG index of attention, which index can be used 
5 to assess his or her mental engagement in a task. 

In accordance with at least one embodiment of the present 
invention, the disclosed apparatus and methods can be used 
specifically for entertainment purposes. For example, a 
player might simply use the system for the enjoyment and 
to challenge of mastering a game or simulation, as well as the 
satisfaction of controlling his own physiological states. It is 
also within the scope of the present invention that a player 
could entertain himself by personally using data feedback to 
set challenges, thresholds and goals for the player to achieve. 
15 Further, it is within the scope of the present invention to 
utilize a variety of input devices or controllers, as well as to 
utilize not just joystick or gamepad input ports, but to utilize 
a variety of other computer input ports as well, and thus 
varying the connection apparatus, as appropriate, to accom- 
20 modate the type of port and/or system being utilized. The 
present invention could possibly utilize interface devices 
such as touch pads, light pens, power gloves, keyboards and 
weapons (e.g., hand-held guns), for example. Additionally, it 
should be noted that when the term “counts” has been used 
25 herein the term potentially referred to any one of a variety of 
types or forms of information that a computer, video game 
system, and/or training simulator mi ght receive at a computer 
or game port. For example, it is within the scope of the present 
invention that the information that the computer software 
30 reads from a port could be in the form of a digital value which 
can be expressed as a number, counts, etc. 

Additionally it is expressly within the scope of the present 
invention that substantially all of the described hardware 
known to the art, for example as described in U.S. Pat. No. 
35 6.450,820, that is incorporated in its entirety by reference 
herein, could be replaced with software. That is to say, for 
example, many functions, such as comparators, timers, 
threshold detectors, relays, voltage controlled amplifier, volt- 
age controlled modulator, offset circuitry, AND gates, VCO, 
40 baseline deviation deriving system, etc., can be accomplished 
with software. For example, these functions could possibly be 
programmed in what is commonly known as a “virtual instru- 
ment” environment, such as National Instruments’ Lab View. 
In at least one embodiment, this type of environment would 
45 allow hardware used in the present invention to be replaced by 
functional modules represented as icons on a screen, which 
can be set-up and “wired” together. The resulting software 
system can then function just like its hardware counterpart, 
for example, interacting with external signals through analog 
50 and digital input and output boards installed in the computer. 

The present invention fully integrates biofeedback training 
into a true dynamic video game, or realistic simulation, which 
turns the popular home pastime of playing video games into 
an opportunity for therapy or self-improvement. It is the 
55 prototype of a new generation of computer game environ- 
ments that train valuable mental skills beyond eye-hand coor- 
dination. 

Current systems typically deliver biofeedback in bland, 
minimally motivating task formats with direct feedback. The 
60 present invention’s video game or task challenge format 
motivates trainees to participate in and adhere to the training 
process through the rewards inherent in mastery of popular 
video games or job simulators, and without the demand, 
monotony or frustration potential of direct concentration on 
65 physiological signals. 

In accordance with at least one embodiment of the present 
invention, the entertainment value and social interaction 
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experience of electronic gameplay is enhanced by distribut- 
ing the control and modulation of inputs to electronic games 
among two or more players, so that joint game goals are 
collaboratively pursued and accomplished by separate play- 
ers who provide different means of control and modulation. 5 
For example, one or more player may provide physical activ- 
ity control via game controller(s) (these players may be 
termed the physical operators) and one or more other indi- 
viduals influencing the game through physiological activity 
measured via body sensors (these players may be termed to 
physiological operators). 

Such an embodiment avoids the problem of movement 
disruption of physiological sensing by modulating one play- 
er’s game controller using the physiological signals of 
another, collaborating player who is physically inactive. The 15 
invention further enables collaborative team play by multiple 
players with different roles on the team — one or more pro- 
viding physiological mastery to facilitate game performance 
and one or more others simultaneously providing manual or 
physical skills needed to operate the game. This creates richer 20 
and more complex gaming opportunities than present games 
provide, enabling engaging and rewarding team social inter- 
actions among people who have different skill sets and inter- 
ests or who take turns playing different roles on a team — on 
one hand, providing the physiological self-mastery, and on 25 
the other, providing physical performance action skills. 

Such an embodiment modulates one or more players’ game 
controller (i.e., game input device operated by physical activ- 
ity) using the physiological signals of another/other, collabo- 
rating player(s) who is/are not providing input to the game via 30 
a game controller. The invention accomplishes this function 
by transmitting control signals, either wired or wirelessly, 
derived from one or more player’s/players’ physiological 
signals to modulate the control over the game of one or more 
other player/players who are using electronic game control- 35 
lers, in a way that either retards or boosts game performance 
influence of the game controllers. 

Alternatively, this shared influence over a game of a physi- 
ological operator and physical (game controller) operators, 
respectively, can be implemented by blending influences 40 
from both into a single control input into the game; in this 
manner, game action tasks could be operated either via a 
game controller or particular predefined physiological input, 
or through pre-defined combinations of both. For instance, a 
video game task of an avatar game character jumping over a 45 
canyon could be initiated by either the game controller opera- 
tor or by a physiological operator’s particular body activity, 
and the height or length of the jump could then be solely 
dependent on the physiological activity of the physiological 
operator whereas the direction of the jump and place of land- 50 
ing on the other side could be controllable only by the game 
controller input. Thus, the physiological and game controller 
operators will have to collaborate to succeed at the task of 
jumping for good success in the game. 

Such an embodiment may be implemented in any of the 55 
above-described game systems, simply by having one play- 
er’s physiological signal(s) sensed by the Physiological Sig- 
nal Conditioning System (e.g., 104 of FIG. 1 ofU.S. Pat. No. 
6,450,820) and having another player control the game con- 
trollers) (e.g., joystick 105 of FIG. 1 ofU.S. Pat. No. 6,450, 60 
820). 

Such an embodiment enables two new categories of elec- 
tronic game play: (1) modulating one or more player’s game 
controller input using the physiological signals of another/ 
other collaborating individuals; and (2) modulating game 65 
activity through joint authority over game actions by game 
controller input from one or more individuals and physiologi- 
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cal sensor input from other separate individuals not using 
game controllers. The embodiment enables physiological and 
manual control of electronic games to be allocated in various 
combinations to several players. 

Such an embodiment permits individuals who are physi- 
cally challenged to participate in electronic game play by 
collaborating with a player who is able to manipulate controls 
that the challenged player cannot, and enables individuals 
with different skill sets and interests (physiological self-con- 
trol vs. physical performance skills) to join together in 
rewarding game play. 

Such an embodiment also enables implementation of 
unprecedented electronic games that incorporate a classic 
motif in superhero comics and movies in which differently - 
abled protagonists collaborate and/or interact, such as Pro- 
fessor X and his cohorts in the X-Men series or Elijah Price 
and David Dunn in the movie “Unbreakable” and provides 
opportunities for new kinds of games where the application of 
“mental powers” of particular individuals (i.e., particular 
deliberately produced EEG characteristics) can enable or 
enrich the performance of other individuals playing a game. 

All references, including publications, patent applications, 
and patents, cited herein are hereby incorporated by reference 
to the same extent as if each reference were individually and 
specifically indicated to be incorporated by reference and 
were set forth in its entirety herein. 

The use of the terms “a” and “an” and “the” and similar 
referents in the context of describing the invention (especially 
in the context of the following claims) are to be construed to 
cover both the singular and the plural, unless otherwise indi- 
cated herein or clearly contradicted by context. The terms 
“comprising,” “having,” “including,” and “containing” are to 
be construed as open-ended terms (i.e., meaning “including, 
but not limited to,”) unless otherwise noted. Recitation of 
ranges of values herein are merely intended to serve as a 
shorthand method of referring individually to each separate 
value falling within the range, unless otherwise indicated 
herein, and each separate value is incorporated into the speci- 
fication as if it were individually recited herein. All methods 
described herein can be performed in any suitable order 
unless otherwise indicated herein or otherwise clearly con- 
tradicted by context. The use of any and all examples, or 
exemplary language (e.g., “such as”) provided herein, is 
intended merely to better illuminate the invention and does 
not pose a limitation on the scope of the invention unless 
otherwise claimed. No language in the specification should be 
construed as indicating any non-claimed element as essential 
to the practice of the invention. 

Preferred embodiments of this invention are described 
herein, including the best mode known to the inventors for 
carrying out the invention. Variations of those preferred 
embodiments may become apparent to those of ordinary skill 
in the art upon reading the foregoing description. The inven- 
tors expect skilled artisans to employ such variations as 
appropriate, and the inventors intend for the invention to be 
practiced otherwise than as specifically described herein. 
Accordingly, this invention includes all modifications and 
equivalents of the subject matter recited in the claims 
appended hereto as permitted by applicable law. Moreover, 
any combination of the above-described elements in all pos- 
sible variations thereof is encompassed by the invention 
unless otherwise indicated herein or otherwise clearly con- 
tradicted by context. 
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The invention claimed is: 

1. A method for modifying the effect of an operator con- 

trolled input device on an interactive device to encourage the 
self-regulation of at least one physiological activity by a 
person different than the operator, the interactive device com- 5 
prising a display device for showing depicted images and a 
reception device for receiving at least one input from the 
operator controlled input device to thus permit the operator to 
control and interact with at least some of the depicted images, 
said method for modifying comprising the steps of: 10 

measuring at least one physiological activity of a person 
different than the operator, to obtain at least one physi- 
ological signal having a value indicative of the level of 
the at least one physiological activity; and 

modifying the ability of the operator to control and interact 15 
with at least some of the depicted images by modifying 
the input from the operator controlled input device in 
response to changes in the at least one physiological 
signal. 

2. The method of claim 1, further comprising the step of 20 
using the at least one physiological signal to control and 
interact with at least some of the depicted images. 

3. The method of claim 1, wherein the operator is a physical 
operator and wherein the person different from the operator is 

a physiological operator. 25 

4. The method of claim 3, wherein the physiological opera- 
tor does not provide direct input to the interactive device via 
an operator controlled input device. 

5. The method of claim 3, wherein the operators interact 
with electronically enhanced communication selected from 30 
the group comprising audio, video, text, and combinations 
thereof. 

6. The method of claim 3, wherein the physiological opera- 
tor is wirelessly connected to the interactive device. 

7. The method of claim 3, further comprising the step of 35 
providing an alternating ability for the operator to take turns 
predominantly as a physical operator or predominantly as a 
physiological operator. 

8. The method of claim 1, wherein the physiological signal 

is selected from the group consisting of EEGs, skin tempera- 40 
ture, skin conductance, heart rate, and combinations thereof. 

9. The method of claim 1, wherein the operator is wire- 
lessly connected to the interactive device. 

10. An apparatus for modifying the effect of an operator 
controlled input device on an interactive device, the interac- 4 - 
tive device comprising a display device for depicting dis- 
played images and a reception device for receiving control 
input from the operator controlled input device, wherein the 
control input permits the operator to control and interact with 

at least some of the displayed images, said modifying appa- 50 
ratus comprising: 

a measurement device for measuring at least one physi- 
ological activity of a person different than an operator of 
the operator controlled input device; 
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a conversion device for converting the at least one mea- 
sured physiological activity into at least one physiologi- 
cal signal having a value indicative of the level of said at 
least one physiological activity; and 

an integration device for modifying the operator’s control 
input in response to changes in the at least one physi- 
ological signal, to thus modify the operator’s ability to 
control and interact with at least one of the depicted 
images in response to changes in the at least one physi- 
ological signal. 

11. An apparatus for modifying the effect of an operator 
controlled input device on an interactive device, the interac- 
tive device comprising a display area, means for depicting 
images upon the display area, and means for receiving input 
from the operator controlled input device, the input permit- 
ting the operator to control and interact with at least some of 
the depicted images, said modifying apparatus comprising: 

means for measuring at least one physiological activity of 
a person different than the operator; 

means for converting the at least one measured physiologi- 
cal activity into at least one signal having a value indica- 
tive of the level of said at least one physiological activity; 
and 

means for modifying the input from the operator controlled 
input device prior to the input being received by the 
interactive device; and 

said modifying means comprising means formodifyingthe 
input from the operator controlled input device in 
response to changes in the at least one physiological 
signal, to thus modify the operator’s ability to control 
and interact with at least one of the depicted images in 
response to changes in the at least one physiological 
signal. 

12. A method for modifying the effect of an operator con- 
trolled input device on an interactive device to encourage the 
self-regulation of at least one physiological activity by a 
person different than the operator, the interactive device com- 
prising a display device for showing depicted images and a 
reception device for receiving at least one input from the 
operator controlled input device to thus permit the operator to 
control and interact with at least some of the depicted images, 
said method for modifying comprising the steps of: 

measuring at least one physiological activity of a person 
different than the operator, to obtain at least one physi- 
ological signal having a value indicative of the level of 
the at least one physiological activity; 

modifying the ability of the operator to control and interact 
with at least some of the depicted images by modifying 
the input from the operator controlled input device in 
response to changes in the at least one physiological 
signal; and 

using the at least one physiological signal to control and 
interact with at least some of the depicted images. 



